Selective Knowledge Distillation for Non-Autoregressive Neural Machine Translation

Benefiting from the sequence-level knowledge distillation, Non-Autoregressive Transformer (NAT) achieves great success in neural machine translation tasks. However, existing distillation has side effects, such as propagating errors teacher to NAT students, which may limit further improvements of models and are rarely discussed research. In this paper, we introduce selective by introducing an evaluator select NAT-friendly targets that high quality easy learn. addition, a simple yet effective progressive method boost performance. Experiment results on multiple WMT language directions several representative show our approach can realize flexible trade-off between complexity training data for models, achieving strong performances. Further analysis shows distilling only 5% raw translations help outperform its counterpart trained about 2.4 BLEU.